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ABSTRACT :  The  Army  Modeling  and  Simulation  Executive  Council  (AMSEC)  recognized  the  requirement  for  a 
distributed  modeling  and  simulation  (M&S)  capability  across  Army  commands  in  March  2003.  A  2-star  level 
Memorandum  of  Understanding  (MOU)  among  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC),  U.S. 
Army  Test  and  Evaluation  Command  (ATEC),  and  U.S.  Amy  Research,  Development  and  Engineering  Command, 
(RDECOM)  formally  documents  this  requirement  in  July  2003.  The  DUSA  (OR)  tasked  the  PM  UA  M&S 
Management  Office  (MSMO)  to  ensure  compatibility  among  the  respective  M&S  capabilities  of  TRADOC, 
RDECOM,  ATEC,  and  the  FCS  Lead  Systems  Integrator  (LSI)  in  order  to  support  concept  exploration,  systems 
integration,  analysis,  and  acquisition  of  the  FCS  Brigade  Combat  Team  (BCT)  System-of-Systems  (SoS).  This 
initiated  the  creation  of  an  Army  M&S  and  data  environment  that  satisfies  the  requirement  for  a  distributed  M&S 
capability  for  all  three  commands  and  the  LSI.  This  initiative  is  defined  as  the  Cross  Command  Collaboration  Effort 
(3CE)  and  is  codified  in  a  supporting  MOA  signed  in  December  2004. 

An  initial  step  in  developing  a  process  to  establish  a  consistent  set  of  tools,  data  and  business  processes  was  the  3CE 
M&S  analysis  conducted  in  August  2005.  This  analysis,  sponsored  by  TRADOC,  used  a  distributed,  live,  virtual,  and 
constructive  (LVC)  environment  to  identify  “best  of  breed”  between  selected  systems  for  inclusion  in  the  3CE 
toolbox.  Battle  command  was  one  of  the  functional  areas  assessed.  The  team  analyzed  two  battle  command  surrogate 
systems.  This  paper  provides  an  overview  of  the  distributed  LVC  environment  along  with  the  methodology  used  to 
conduct  the  analysis,  lessons  learned  and  recommendations  on  how  this  process  should  be  used  to  support  future 
assessments. 


1 


Report  Documentation  Page 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 

1.  REPORT  DATE 

JUN  2006 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2006  to  00-00-2006 

4.  TITLE  AND  SUBTITLE 

Battle  Command  System  Analysis  Methodology  in  the  Cross  Command 
Collaborative  Effort  (3CE)  Environment 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

The  MITRE  Corporation, 7515  Colshire  Drive, McLean, VA, 22102 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

30 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98} 

Prescribed  by  ANSI  Std  Z39-18 


1,  Introduction 

The  Army  Modeling  and  Simulation  Exeeutive  Couneil 
(AMSEC)  reeognized  the  requirement  for  a  distributed 
modeling  and  simulation  (M&S)  eapability  across 
Amy  commands  in  March  2003.  A  2-star  level 
Memorandum  of  Understanding  (MOU)  among  the 
U.S.  Army  Training  and  Doctrine  Command 
(TRADOC),  U.S.  Amy  Test  and  Evaluation 
Command  (ATEC),  and  U.S.  Amy  Research, 
Development  and  Engineering  Command,  (RDECOM) 
fomally  documents  this  requirement  in  July  2003. 
The  DUSA  (OR)  tasked  the  PM  UA  M&S 
Management  Office  (MSMO)  to  ensure  compatibility 
among  the  respective  M&S  capabilities  of  TRADOC, 
RDECOM,  ATEC,  and  the  PCS  Lead  Systems 
Integrator  (LSI)  in  order  to  support  concept 
exploration,  systems  integration,  analysis,  and 
acquisition  of  the  PCS  Brigade  Combat  Team  (BCT) 
System-of-Systems  (SoS).  This  initiated  the  creation 
of  an  Amy  M&S  and  data  environment  that  satisfies 
the  requirement  for  a  distributed  M&S  capability  for 
all  three  commands  and  the  LSI.  This  initiative  is 
defined  as  the  Cross  Command  Collaboration  Effort 
(3CE)  and  is  codified  in  a  supporting  MOA  signed  in 
December  2004. 

2,  The  3CE  Environment 

End  state  for  3CE  is  the  development  of  a  cross 
command  Army  M&S  and  data  environment  used  for 
design,  development,  integration,  and  testing  of 
capabilities,  systems,  and  prototypes.  3CE  intends  to 
provide  a  set  of  common  and  consistent  M&S  tools, 
data,  and  business  processes  used  by  TRADOC, 
ATEC,  RDECOM,  and  Program  of  Record  Manager  in 
order  to  allow  the  Amy  to  develop  concepts, 
prototypes,  and  test  and  evaluation  methodologies 
using  consistent  processes. 

To  achieve  the  end  state  described  above  and  to  enable 
a  near-term  utilization  of  an  evolving  and  maturing 
3CE  environment,  3CE  objectives  must  encompass  two 
perspectives:  event  execution  and  capability 

development.  Event  execution  objectives  must  enable 
an  M&S  environment  to  support  the  3CE  supported 
program  of  record  using  existing  capabilities  available 
in  TRADOC ’s  Battle  Lab  Collaborative  Simulation 
Environment  (BLCSE),  ATEC’s  Distributed  Test 
Environment  (DTE),  and  RDECOM’s  Modeling 
Architecture  for  Technology  Research  and 

Experimentation  (MATREX).  The  capability 

development  objectives  must  enable  a  collaborative 


effort  to  identify,  define,  and  develop  a  core  set  of 
M&S  tools,  data,  and  business  processes  that  satisfy 
the  common  required  environment  capabilities  of  the 
three  Commands  and  the  3CE  supported  program  of 
record’s  materiel  developer. 

3,  Purpose  of  the  M«&S  Analysis 

The  purpose  of  the  M&S  analysis  was  to  begin  the 
identification  and  development  of  the  3CE  toolbox 
consisting  of  M&S  systems,  surrogates,  models,  and 
tools  with  specific  capabilities  to  meet  force 
development  requirements.  The  initial  step  in  this 
process  was  to  develop  a  thorough  understanding  of 
the  systems.  The  process  included  identifying  the 
requirement  for  which  the  system  was  designed  to 
satisfy,  the  systems’  capabilities  and  limitations,  and 
redundant  capabilities  between  systems.  This 
information  enables  a  decision  on  whether  or  not  a 
particular  M&S  system,  surrogate,  or  tool  should  be 
included  in  the  3CE  tool  box.  Additionally,  this 
information  enables  a  “best  of  breed”  decision  between 
redundant  systems. 


n  Common  M&S  and  Data  Requirements 


Program  of  Record 
ATEC  Materiel  Developer 


O  Unique  M&S  and  Data  Requirements 

Figure  3.1 

An  initial  screening  of  M&S  systems  available  to  3CE 
identified  several  systems  that  appeared  to  perform 
similar  functions.  This  apparent  redundancy  initially 
focused  the  M&S  analysis  work  on  two  functional 
areas:  communications  and  battle  command.  These 
functional  areas  satisfied  the  shared  space  for  the 
supported  3CE  members  as  depicted  in  the  “common” 
M&S  requirements  area  of  Figure  3.1.  Therefore,  the 
M&S  analysis  comprised  communications  effects  tools 
and  battle  command  system  (BCS)  surrogates.  Since 
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the  focus  of  this  conference  is  on  battle  command 
related  issues,  the  remainder  of  this  paper  will  be 
limited  to  the  battle  command  system  surrogate 
analysis. 

4,  Future  Force  Battle  Command 

Battle  Command  is  the  art  and  science  of  applying 
leadership  and  decision-making  to  achieve  mission 
success.  *  Future  Forces  will  be  enabled  by  Networked 
Battle  Command  which  seeks  a  more  holistic  balance 
between  art  and  science  and  between  deliberate 
planning,  leadership,  and  decision-making.  Previous 
approaches  over-emphasized  the  science  and  deliberate 
planning  aspects  of  Battle  Command  at  the  expense  of 
the  art,  leadership,  and  decision-making  aspects.  This 
approach  inadvertently  led  to  command  posts  that 
tethered  the  commander  to  that  location,  to  an  absence 
of  “on-the-move”  capabilities,  and  to  multiple  stove- 
piped  systems  that  aided  decision  making  in  a  narrow 
view.  Two  surrogate  battle  command  systems  were 
selected  for  this  initial  3CE  analysis:  a  future  force 
battle  command  system  surrogate  and  a  current  force 
battle  command  system  surrogate.  Within  this 
Networked  Battle  Command  environment  which  our 
future  forces  will  operate  and  due  to  this  systems  of 
systems  approach,  it  is  critical  to  have  a  consistent 
battle  command  system  when  doing  concept 
exploration  and  making  acquisition  decisions.  If  this 
does  not  happen,  it  is  possible  that  a  system  conducting 
successful  concept  exploration  in  one  command  would 
not  achieve  success  during  testing  in  another 
command. 

4,1  Analytic  Methodology 

There  were  three  phases  to  the  3CE  toolkit  analysis. 
The  focus  of  Phase  I  was  to  identify  users’ 
requirements  for  the  M&S  components  and  the  tools 
under  analysis.  The  analysis  team  solicited  inputs  from 
the  commands  that  used  the  respective  battle  command 
system  surrogates.  The  requirements  provided  a  way 
to  compare  and  contrast  the  systems  under  analysis. 
The  like  requirements  provided  a  way  to  compare  the 
systems  and  the  unique  requirements  provided  a  way  to 
contrast  the  systems  (Figure  4.1).  This  was  the  basis 
for  the  analytic  methodology.  The  requirements  were 
developed  into  technical,  functional,  and  operational 
metrics.  These  metrics  provided  the  means  to  analyze 
the  capabilities  of  the  battle  command  system 
surrogates.  The  end  product  of  this  phase  was  the  draft 


Data  Collection  and  Management  Plan  (DCMP)  that 
laid  out  the  measures  of  merit  (MoM)  and  data  element 
requirements  for  the  metrics. 


I  Like  Requirements 
'  basis  for  comparison 

System  A  [  / 

1  Requirements! 

System  B  \ 
Requirements  / 

'  UNIQUE  requirements  ' 
basis  for  contrast 

Figure  4.1 


During  Phase  II,  the  analysis  team  assessed  the  data 
generation  and  data  collection  capabilities  of  the 
distributed  LVC  event  to  satisfy  the  metrics  and 
finalized  the  analytic  approach.  During  this  phase,  the 
analysis  team  concluded  that  it  was  infeasible  to  assess 
the  operational  metrics.  The  rationale  for  this  decision 
is  discussed  in  the  Mission  Threads  section  of  this 
paper.  The  purpose  of  Phase  III  was  to  analyze  the 
battle  command  system  surrogates  during  the  Spirals 
leading  up  to  the  event  and  during  the  conduct  of  the 
event. 

The  execution  concept  of  operations  to  support  the 
analysis  is  depicted  in  Figure  4.2.  The  analysis  team 
collected  the  technical  data  before  the  distributed  LVC 
event  or  spirals.  This  data  consisted  primarily  of  the 
system  specifications.  The  analysis  team  collected  the 
functional  data  during  the  spiral  test  events.  This 
process  equates  to  a  bench-test  and  it  enabled  a  better 
understanding  and  validation  of  the  capabilities  of  the 
system.  Finally,  the  analysis  team  evaluated  the 
systems  under  load,  in  the  context  of  the  distributed 
LVC  environment.  This  process  equates  to  a  field-test 
and  it  enables  analysis  of  the  system  within  the  context 
of  its  intended  use.  While  the  analyzed 
communications  systems  were  distributed,  both  Battle 
Command  surrogate  systems  were  co-located.  Subject 
matter  experts  (SME)  for  the  systems  were  co-located 
with  the  systems  and  available  for  consultation  with 
the  analysis  team. 


'  FCS  ORD  w/  Change  3  (JROC  Approved),  dated  14 
April  2003. 
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Figure  4.2 


The  analysis  team  developed  and  sent  a  questionnaire 
to  the  respective  commands  to  identify  the  business 
processes,  policies,  and  procedures  for  using  these 
systems.  Data  gathered  by  the  SMEs  were  combined 
with  the  questionnaire  responses  to  assess  the  technical 
and  functional  capabilities  of  each  Battle  Command 
surrogate  system.  The  plan  was  to  combine  these  SME 
and  questionnaire  results  with  federation  output  data  to 
support  the  operational  analysis.  Flowever,  the 
conduct  of  the  event  did  not  represent  a  dynamic 
operational  environment.  Consequently,  federation 
output  data  could  not  be  used  to  analyze  the 
operational  capability  of  the  systems. 

4.2  Scope  of  Comparisons 

The  scope  of  the  3CE  M&S  analytic  effort  was  limited 
to  two  functional  areas:  Battle  Command  and 
Communications  Effects.  The  analysis  team  analyzed 
two  systems  within  each  of  these  functional  areas.  As 
previously  noted,  this  paper  focuses  on  the  area  of 
Battle  Command.  The  focus  of  the  analysis  was  to 
gather  data  to  address  the  issues  listed  in  the  3CE 
Analysis  Plan,  using  the  Battle  Command  DCMP.  The 
DCMP  identified  metrics,  measures,  data  and  data 
sources  for  the  data  needed  to  analyze  the  systems. 
The  Analysis  Plan  and  the  DCMP  were  updated 
following  Phases  I  and  II  to  ensure  that  the  previously 
identified  analytic  approach  was  valid  and  within  the 
scope  of  the  distributed  LVC  event. 

4.3  Mission  Threads 

The  analysis  team  identified  a  series  of  mission  threads 
to  support  the  operational  portion  of  the  Battle 
Command  System  surrogate  analysis.  These  threads 
included  Intelligence,  Situational  Awareness,  Fires  and 
Sustainment.  The  distributed  LVC  event  used  a  Time 
Ordered  Event  List  (TOLL)  to  synchronize  and  control 
the  event.  The  analysis  team  reviewed  the  TOLL  to 
determine  which  mission  threads  could  potentially  be 


represented  during  the  event.  Mission  threads  enable 
the  analysis  of  the  operational  metrics  in  the  Battle 
Command  DCMP.  The  analysis  team  then  cross- 
walked  the  TOEL  mission  threads  with  the  Battle 
Command  DCMP  in  order  to  identify  the  specific 
thread  or  threads  applicable  to  each  metric  in  the 
DCMP.  The  analysis  team  attempted  to  collect  data  on 
some  of  these  threads  during  and  following  the 
distributed  LVC  spiral  events.  A  review  of  the  output 
databases  resulted  in  a  conclusion  that  the  collection  of 
the  operational  data  required  to  support  the  analysis 
was  infeasible.  The  primary  reason  for  this  conclusion 
was  the  fact  that  the  tactical  events  could  not  be  cross- 
walked  with  the  entities  associated  with  a  mission 
thread.  Furthermore,  the  static  nature  of  a  TOEL 
sequenced  event  does  not  support  the  dynamic 
operational  conditions  required  to  properly  analyze  a 
Battle  Command  System  surrogate. 

4.4  Sources  of  Data 

The  preferred  source  for  data  was  from  the  M&S 
federation,  including  the  components  under  analysis 
during  scenario  or  vignette  execution.  The  analysis 
team  worked  with  the  scenario  development  team  to 
ensure  that  the  scenario/vignettes  created  the  proper 
context  to  support  the  comparisons.  Execution  of  the 
scenario/vignettes  via  the  TOEL  did  not  support  a 
dynamic  operational  environment;  therefore,  federation 
output  data  did  not  support  the  M&S  analysis  along 
mission  threads.  The  analysis  team  relied  on  technical 
and  functional  data  for  analysis.  Functional  testing, 
SME  interviews,  and  questionnaires  were  the  primary 
sources  of  data  for  the  M&S  analysis.  The  Battle 
Command  system  surrogate  SMEs  were  habitual  users 
and  event  operators  of  the  systems  and  technicians  for 
the  respective  systems. 

4.5  Comparison  Environment 

The  Battle  Command  System  surrogates  were  part  of  a 
distributed  LVC  environment.  As  shown  in  Figure  4.3, 
the  environment  comprised  two  High  Level 
Architecture  (HLA)  federations  and  several  Distributed 
Interactive  Simulation  protocol  (DIS)  systems.  There 
were  65  unique  federates  and  130  total  federates.  The 
LVC  components  were  distributed  among  various 
nodes/sites  across  the  United  States.  Gateways 
translated  DIS  and  HLA  messages  and  a  federation-to- 
federation  bridge  linked  the  two  HLA  federations. 
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4,6  Summary  of  Analytic  Results 

There  were  minimal  discemable  technieal  differenees 
between  the  two  battle  command  components.  The 
primary  technical  difference  deals  with  the  mechanism 
each  uses  to  exchange  data  within  a  simulation-driven 
federation.  One  system  used  the  DIS  protocol  to 
exchange  information,  while  the  other  system  used 
HLA. 

Table  1  provides  a  summary  of  the  battle  command 
comparison  results  from  a  functional  standpoint  by 
summarizing  the  specific  SME  ratings  based  upon  the 
MoM  defined  in  the  DCMP.  In  this  table,  each  system 
was  given  a  Green  (G),  Amber  (A)  or  Red  (R),  or  a 
combination  (e.g.  G/A  or  A/R),  rating  based  on  its 
ability  to  satisfy  the  study  issues  and  corresponding 
MoMs.  A  rating  of  Green  indicates  the  system  fully 
met  all  of  the  MoMs  for  that  area.  A  rating  of  Amber 
indicates  the  system  met  the  MoM  but  not  fully.  A 
rating  of  Red  means  that  the  system  did  not  have  that 
capability.  A  combined  rating  such  as  G/A  or  A/R 
means  that  the  system  partially  met  some  of  the  areas 
while  not  having  capabilities  in  others.  As  stated  in 
previous  sections,  the  SMEs  consisted  of  a 
combination  of  operators  and  technicians  with  the  lead 
SME  making  the  final  determination  of  the  rating. 

Overall,  BCSl  met  more  of  the  battle  command 
surrogate  requirements  identified  for  the  comparison. 
This  is  not  surprising  since  the  primary  source  of 
requirements  for  this  examination  and  the  development 
of  BCSl  was  based  upon  Future  Force  BCS 
requirements.  BCS2  was  designed  to  replicate  Current 
Force  capabilities  in  support  of  Army  testing 
requirements  (i.e.  FBCB2).  The  team  was  unable  to 
develop  a  more  robust  data  collection  capability  to 


examine  BCS2’s  unique  capabilities  since  the 
development  requirements  for  BCS  2  were  not  readily 
available  to  the  analysis  team.  The  one  exception  was 
the  requirement  to  send  and  receive  tactical  messages 
(USMTF  and  JVMF).  BCS2  performed  those 
functions  well  while  BCSl  did  not  have  this  capability. 


Table  1:  Functional  Area  Results 


Area  of  Examination 

BCS1 

BCS2 

COP  Functionality 

G 

A/G 

Intelligence  Functionality 

A/R 

R 

Fires  &  Effects  Functionality 

A/R 

R 

C2  Functionality 

A/R 

A/G 

Collaboration  Functionality 

A/R 

R 

Mob/CM/Surv  Functionality 

A/R 

A/R 

Sustainment  Functionality 

A/R 

A/R 

Maneuver  Functionality 

A/R 

A/R 

Training  Functionality 

R 

A/R 

Stimulate  Tactical  Systems 

R 

G 

5,  Lessons  Learned 

As  a  result  of  executing  the  M&S  analysis,  there  were 
numerous  lessons  learned  that  can  be  categorized  as 
process  improvements  and  technical  challenges. 

5,1  Process  Improvements 

With  respect  to  the  process,  the  three  phased  approach 
for  the  analytic  methodology  should  be  maintained; 
although  there  is  room  for  improvement  across  all  the 
phases.  During  planning,  the  systems  under  analysis, 
as  well  as  the  required  capabilities,  must  be  identified 
up  front  and  have  the  support  of  all  participants.  Roles 
and  responsibilities  associated  with  the  analysis  must 
be  clearly  defined,  and  care  must  be  exercised  in  the 
selection  of  systems  for  analysis  to  avoid  examining 
dissimilar  systems.  All  user  requirements  for  each  of 
the  functional  areas  must  be  identified  up  front.  Since 
this  is  the  basis  for  the  analytic  metrics,  this  is  critical 
to  the  success  of  the  analysis.  Without  a  clear 
identification  and  understanding  of  the  user 
requirements  for  which  the  system  was  intended  to 
satisfy,  one  cannot  achieve  an  acceptable  assessment  of 
the  systems.  If  all  of  the  users  are  not  represented  by 
the  set  of  requirements  utilized  to  derive  the  metrics  for 
the  analysis,  there  may  be  analytic  bias.  Furthermore, 
the  analysis  must  account  for  unique  user  requirements 
to  highlight  the  unique  and  often  user  specific 
capabilities  that  the  system  provides  the  user  in 
addition  to  common  requirements  that  provide  the 
basis  for  comparing  systems.  During  Phase  I, 
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identifying  valid  user  requirements  from  documented 
sources  and  gaining  the  consensus  of  the  users  on  the 
metrics  for  the  analysis  is  paramount.  To  ensure  the 
environment  supports  the  analysis,  operational 
requirements  (mission  threads)  must  be  identified  early 
and  data/  scenario  requirements  must  be  coordinated 
with  the  applicable  working  groups. 

During  Phase  II,  the  Analysis  team  must  verify  that  the 
data  collection  mechanisms  and  data  repository 
structures  are  sufficient  to  meet  DCMP  requirements. 
They  must  also  verify  that  the  event  environment  will 
provide  the  means  to  employ  the  system  under  analysis 
within  the  context  for  which  it  was  intended  to  be  used. 
These  are  prerequisites  to  initiating  Phase  III.  This 
validation  of  the  analytic  approach  requires  the 
involvement  of  all  user  representatives,  to  include  all 
stakeholders  agreeing  to  the  scenario  and  ensuring  it 
supports  the  analytic  approach. 

During  Phase  III  adequate  time  and  resources  must  be 
dedicated  to  the  technical,  functional  and  operational 
testing  outlined  in  the  DCMP.  Also  the  data  collected 
and  archived  in  databases  must  be  properly  formatted, 
readily  available,  and  usable  by  the  analyst  to  support 
post-event  analysis  and  reporting  requirements. 

5.2  Technical  Challenges 

The  fundamental  technical  lesson  learned  is  a  recurring 
one  for  many  complex  federations.  Insufficient  time 
and  resources  are  allocated  for  integration  testing  and 
analytic  data  collection  verification.  If  an  event  is 
executed  in  spite  of  missing  critical  testing  gates,  the 
environment  is  often  technically  unstable  or  provides 
sporadic  operability,  especially  in  complex 
environments.  Under  these  conditions,  it  is  nearly 
impossible  to  ascertain  and  isolate  which  systems  work 
and  how  they  perform.  The  architecture  and 
components  must  be  fully  operational  during  testing. 
For  example,  during  the  distributed  LVC  event  used  in 
the  conduct  of  this  analysis,  there  were  several 
technical  difficulties  that  were  not  identified  until  the 
conduct  of  the  event.  This  was  a  result  of  incomplete 
end-to-end  testing  during  the  spiral  events  leading  up 
to  the  event.  These  technical  issues  resulted  in  an 
incomplete  assessment  of  the  battle  command  systems. 
Also,  when  linking  FILA  and  DIS  federations  there  are 
significant  challenges  that  arise  that  cause  the  data  trail 
to  be  difficult  to  follow  (i.e.,  different  FILA  and  DIS 
entity  IDs  for  the  same  entity).  This  entity  ID 
inconsistency,  along  with  federation  components 
periodically  going  down,  left  gaps  in  the  data  and  did 
not  support  the  analytic  requirements.  Along  these 
same  lines,  there  must  be  a  coordinated,  well  integrated 


database  system  designed,  developed,  and  tested  prior 
to  execution.  The  data  repository  used  during  the 
distributed  LVC  event  segregated  the  tactical,  DIS  and 
two  FILA  federation  messages  and  there  was  no 
straight  forward  way  to  integrate  and  synchronize 
those  messages.  This  made  it  extremely  difficult  and 
time  consuming  to  conduct  end-to-end  mission  thread 
analysis.  The  data  repository  must  be  centralized  or 
must  provide  a  means  to  synchronize  results,  while 
providing  all  users  with  remote  access  and  shared 
products  and  services.  In  order  for  the  data  to  support 
analysis,  consistent  time  stamping  and  entity 
identification  mechanisms  across  the  federation  are 
required. 

Lastly,  the  varied  business  practices  of  the 
organizations  involved  in  this  event  required  detailed 
upfront  planning  to  reconcile  these  differences.  For 
example,  some  agencies  required  a  rigid  TOLL  to 
support  their  analysis  while  other  agencies  required  a 
more  dynamic  operational  environment  for  analysis. 
The  M&S  analysis  described  in  this  paper  requires 
both  methods  in  order  to  support  the  functional  bench¬ 
testing  and  the  operational  mission  thread  analysis.  If 
the  event  designers  do  not  understand  or  account  for 
these  analytic  requirements,  the  conduct  of  the  event 
will  not  support  the  analysis,  which  is  the  purpose  of 
conducting  the  event. 

6.  Recommendations 

As  the  3CE  environment  matures,  there  will  be  a  need 
for  future  M&S/  toolkit  analyses.  These  and  other 
types  of  cross-command  analyses  must  be  well  defined 
and  all  stakeholders  must  have  a  thorough  and 
complete  understanding  of  the  requirements.  This 
includes  well  defined  data  elements.  As  part  of 
defining  these  requirements  early,  the  selection  of 
systems  for  analysis  should  include  all  similar 
models/tools  to  establish  a  robust  comparative  analysis 
environment.  The  analytic  methodology  of  analyzing 
the  M&S  systems  and  tools  from  a  technical, 
functional  and  operational  perspective  should  be 
maintained  and  improved  upon,  while  ensuring  that  the 
architecture,  scenario  and  data  collection  requirements 
satisfy  the  analytical  objectives.  During  the  technical 
and  functional  portions  of  the  analysis,  a  static  or 
bench-testing  environment  for  each  system  is 
preferred.  During  the  operational  phase,  each  system 
should  perform  in  its  “normal”  or  intended  operating 
environment  under  various  conditions  of  load  to 
demonstrate  its  functionality,  reliability,  and  stability. 
The  scenario  and  data  collection  schema  must  support 
the  comparative  analyses  by  setting  the  conditions  for 
the  analysis  team’s  analysis.  Lastly,  the  stability  of  the 
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federation  must  be  improved  in  order  for  the  analysis 
team  to  understand  and  isolate  cause  and  affect 
activities  in  the  scenario  relative  to  the  systems  under 
analysis  and  to  capture  usable  data  to  support  the  final 
analysis.  SMEs  must  be  resourced  to  be  in  the  correct 
place  at  the  correct  time  to  provide  objective  support  of 
the  analytic  effort. 

A  few  aspects  that  were  not  specifically  addressed  in 
the  DCMP,  but  are  critical  to  consider  during  a 
analysis  such  as  this  are  cost,  user  training,  and  a 
vision  of  the  future  for  the  M&S  system.  Costs  not 
only  include  hardware,  software,  licensing,  and 
integration  (all  were  accounted  for  in  the  technical 
portion  of  the  DCMP  for  this  analysis),  but  cost 
consideration  must  also  account  for  the  sunk  costs  of 
programs  in  development  and  training  costs  of 
bringing  a  new  system  into  the  environment.  More 
importantly  are  the  costs  associated  with  improving  a 
system  selected  during  a  “best  of  breed”  M&S  analysis 
to  account  for  capability  gaps  previously  covered  by 
the  non-selected  system.  Since  it  may  require 
significant  resources  to  improve  even  limited 
functionality  deficiencies  in  an  M&S  system,  the  cost 
for  these  improvements  should  be  closely  examined 
and  should  be  one  of  the  key  parameters  considered 
prior  to  selecting  one  system  over  another.  Lastly,  is 
the  consideration  of  “how  does  the  model  or  tool  fit 
with  the  future  vision  of  the  environment?”  One 
should  not  be  short-sited  in  planning  and  developing 
an  M&S  and  data  environment  and  should  consider  the 
long  range  goal  of  the  environment  in  the  conduct  of 
the  M&S  analysis. 

7.  Conclusions 

This  initial  3CE  M&S  analysis  provides  a  process  and 
methodology  for  future  efforts.  Although  the  original 
intent  of  this  comparison  was  to  determine  a  “best  of 
breed”,  it  was  realized  that  these  systems  were 
developed  for  their  respective  sponsors/commands  for 
a  specific  purpose.  Consequently,  the  unique  user 
requirements  for  each  of  the  systems  made  it  infeasible 
to  select  one  system  to  fulfill  both  missions. 
Furthermore,  to  recommend  a  “best  of  breed”  without 
fully  realizing  the  implications  due  to  the  inability  to 
completely  evaluate  the  systems  properly  would  have 
been  ill  advised.  This  effort  did  provide  insights  on  a 
methodology  and  executable  process,  and  just  as 
importantly,  provided  a  better  understanding  of  the 
capabilities  and  limitations  of  the  systems  analyzed  in 
this  environment,  as  well  as  the  dynamics  of 
conducting  a  distributed  LVC  event  with  several 
agencies  with  distinct  and  often  varied  methods  of 
doing  business.  It  also  highlighted  how  these  systems 


can  work  together  in  the  future  to  achieve  certain 
objectives  and  how,  with  further  analysis,  systems  can 
be  used  more  efficiently.  With  the  many  initiatives 
occurring  to  develop  the  future  battle  command 
system,  this  process  will  be  able  to  be  implemented  to 
determine  a  consistent  system  within  the  3CE 
environment. 
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Battle  Command  System  Analysis  Methodology  in  the 
Cross  Command  Collaboration  Effort  (3CE) 

Environment 


Command  and  Control  Research  and 
Technology  Symposium 


1  June  2006 


Purpose  of  Brief 


To  use  a  3CE  event  as  an  illustrative  example  of  an  analysis 
methodology  being  developed  to  build  the  3CE  environment. 


Endstate 

To  give  the  audience  an  understanding  of  the  3CE 
environment,  its  challenges  and  recommendations  on  how  to 
determine  what  the  toolkit  should  consist  of. 


“Out  of  intense  complexities,  intense  simplicities  emerge.” 

-Winston  Churchill 


Outline 


*  3CE  Overview 

-  Purpose 

-  Mission  and  intent 

-  Objectives 

-  Significant  Activities 

*  Analysis 

-  Purpose 

-  Approach 

-  Mission  Threads 

-  Execution 

*  Lessons  Learned 

*  Recommendations 


3CE  Purpose 


•  3CE  objective  per  the  MOD  (July  2003): 

-  Maximize  the  rapid  availability  of  transformational 
technology  to  the  field  soldier  by  leveraging  the  synergy 
gained  from  integrating  the  activities  of  each  of  the  three 
commands  into  a  holistic  cooperative  effort. 

•  DUSA  OR  Task  to  PM  PCS  MSMO: 

-  Ensure  compatibility  among  the  respective  M&S 
capabilities  of  TRADOC,  RDECOM,  ATEC,  and  the  PCS  LSI 
in  order  to  support  concept  exploration,  systems 
integration,  analysis,  and  acquisition  of  the  PCS  BCT  SoS. 

•  3CE  purpose  per  the  MOA  (December  2004): 

-  Develop  cross  command  Army  M&S  and  data 
environments  that  will  be  used  in  Systems  of  Systems 
(SoS)  design,  development,  integration,  and  test  of  PCS 
PoS  components,  systems,  and  prototypes  within  a 
realistic  PCS  BCT  context. 


3CE  Mission  and  Intent 


Mission:  Deveiop  a  cross  command  Army  M&S  and  data  environment  for  design, 
deveiopment,  integration,  and  testing  of  capabiiities,  systems,  and  prototypes. 


JCIDS  < 

_ _ Ji 

Concept 
.  Refinement 

'  Concept 
t  Decision 

Technology 

Development 

1  JROC  Pre-Systems  Acquisition 

IOC 


FOC 


System  Development 
&  Den^^stration 

Critir.al  npsign  Rgyipw 


Production  &  Deployment 

FRP 

Decision 


LRIP/IOT&E 


Review 


Operations  & 
Support 


Systems  Acquisition 


Sustainment 


3CE  /  Core  Products 


1  ^ — r  ^ ^ 

Process  & 
Procedures 

Data 

1  Toolbox  1 

^ - ^ 

intent: 

Key  Tasks:  identify,  deveiop,  and  maintain  a  core  set  of  M&S 
toois,  data,  and  business  processes  that  provide  interoperabie 
connectivity  that  iinks  the  participating  organizations, 
to  inciude  providing  a  common  3CE  environment  and  expertise 
for  the  Army  to  ieverage. 


ATEC 


RDECOM 


End  State:  A  3CE  environment  that  meets  the  common  requirements  of  aii  three 
commands  and  PM  PCS  BCT  to  conduct  distributed  DOTMLPF  deveiopment. 


“Effects” 


*  Establish  a  common  problem  analysis,  requirements 
development  and  engineering  methodoiogy  across  the 
three  commands  technical  community. 

*  Establish  a  common  ianguage  and  perspective  of  M&S 
technology  domains. 

*  Deveiop  capabiiities  that  are  traceabie  to  user  needs  and 
design  requirements. 

*  Implement  capabilities  for  the  Army  “to  be”  M&S  and 
data  environment. 


Enable  analysis  and  evaluation  through 
distributed  M&S  LVC  capabiiities. 


3CE  Significant  Activities 


*  Establishing  and  documenting  procedures  for 
development,  control,  and  use  of  3CE. 

*  Maintaining  a  cross-command,  distributed  network 
capabie  of  supporting  a  iive,  virtuai,  and  constructive 
(LVC)  environment  using  existing  capabiiities. 

*  Establishing  a  Systems  Engineering  approach. 

*  Deveioping  a  capabilities  cataiogue. 

*  Establishing  a  support  framework  to  enabie 
interoperability. 


Identifying  analysts  and  evaluators’  needs 
to  drive  technical  development 


Purpose  of  the  Analysis 


•  PURPOSE:  The  purpose  of  the  analysis  was  to  identify  “best  of 
breed”  M&S  tools  for  inclusion  in  the  notional  3CE  tool  box. 

•  METHOD:  Comparative  analysis  of  select  systems  based  upon 
user  requirements.  Analysis  occurred  during  the  spiral  events 
leading  to  DTE5  event,  as  well  as  during  DTE5,  scheduled  for  22 
August  -  2  September  2005. 

•  END  STATE:  Recommendation  of  “best  of  breed”  capabilities  for 
the  notional  3CE  toolbox. 
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Constraints  -  Limitations  -  Assumptions 


•  Study  Constraints 

-  The  two  communications  systems  under  comparison  were  not  co-located. 

-  DTE  5  was  a  time  ordered  event  list  (TOEL)  driven  event  that  did  not  allow  for  dynamic  operations. 

•  Study  Limitations: 

-  The  basic  premise  of  the  M&S  comparison  relies  on  the  assumption  that  M&S  systems  designed  for 
specific  and  different  purposes  (experimentation  and  test)  could  be  compared  on  the  basis  of  similar 
user  requirements.  For  the  most  part,  the  analysis  team  validated  this  assumption.  However  the 
analysis  was  limited  by  the  fact  that  there  were  significant  differences  in  functional  capability  because 
the  respective  commands  did  not  have  the  requirement  to  develop  some  of  the  functionalities 
examined  in  this  comparison. 

-  The  enumeration  mappings  between  the  Distributed  Interactive  Simulation  (DIS)  and  High  Level 
Architecture  (HLA)  federates  were  inconsistent,  which  prevented  the  HLA  unit  icons  from  being 
displayed  correctly  on  the  C2  systems  in  the  DIS  environment. 

-  The  time  available  for  the  M&S  component  comparisons,  given  other  priorities  for  DTE  5  and  the  spiral 
events,  limited  the  observation  portion  of  the  analysis  to  three  days  (30  August  to  1  September  2005). 

-  The  stability  of  the  federation-to-federation  bridge  limited  the  battle  command  data  collection  and 
analysis  effort.  The  battle  command  surrogates  at  UAMBL  lost  all  situational  awareness  (SA) 
information  on  the  common  operating  picture  (COP)  when  the  federation-to-federation  bridge  stopped 
working.  Consequently,  data  collection  ceased  until  the  bridge  was  restored. 

-  The  TOEL-driven  scenario  did  not  support  end-to-end  mission  threads  analysis  thus  preventing  the 
examination  of  the  operational  component  of  the  model  comparison. 

•  Validated  Assumptions: 

-  The  a  priori  assumption  that  there  are  redundant  capabilities  across  the  commands  was  validated  to 
some  extent.  Additional  analysis  is  required  to  determine  which  capabilities  should  be  carried  forward. 

-  The  assumption  that  the  information  gathered  by  the  subject  matter  experts  (SMEs)  and  the 
Commands’  responses  to  the  3CE  Comparison  Questionnaire  would  provide  sufficient  data  to  support 
the  analysis  was  validated  for  the  technical  and  functional  comparisons. 


Analytic  Approach 


•  PHASE  I:  Determine  basis  of  comparison. 

-  Dates:  1  May  2005  -  30  June  2005 

-  Description:  Determine  comparison  requirements  based  upon 
analytical  user  requirements  of  the  specified  tools.  Meet  with 
technical  and  user  representatives  to  develop  knowledge  base. 
Develop  and  document  in  a  data  collection  management  plan 
the  systems,  metrics,  and  data  element  requirements  to  answer 
the  metrics. 

•  PHASE  II:  Verify  data  collection  capability  and  validate  analytical 
approach. 

-  Dates:  1  July  2005  -  19  August  2005 

-  Description:  Verify  data  collection,  finalize  Analysis  Plan,  DCMP, 
test  threads  and  validate  analytical  methodology  during  Spirals 
6  and  7. 

•  PHASE  III:  Conduct  data  collection  and  analysis. 

-  Dates:  20  August  2005  -  30  days  after  DTE5  ENDEX 

-  Description:  Coiiect  data,  conduct  anaiysis,  and  write  the  report. 
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Analytic  Approach 


statement  of  the  Problem 


'\ 


Phase  II:  Finalize  user 
Requirements,  Analysis 
Plan  and  DCMP;  assess  and  ^ 
rehearse  data  collection 
procedures 


Phase  I:  Develop  draft 
Analysis  Plan  and  DCMP, 
begin  to  collect  user 
requirements 


Phase  III:  Execute  DTE5, 
^  collect,  reduce  and  analyze 
data,  and  document  results 
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DCMP  Purpose 


•  Codifies  user  requirements  and  recasts  them  in  analytical 
terms  (issues,  EEAs,  MOMs) 

•  Crosswalks  the  objectives,  issues,  EEAs  and  MOMs  for  the 
Battle  Command  and  Communications  3CE  M&S  component 
comparisons 

•  Identifies  sources  for  data  collection  (model,  SMEs,  Interviews 
or  Surveys) 

•  Identifies  the  context  in  which  the  data  for  each  MOM  will  be 
collected  (standalone  or  using  an  operational  thread  (Fires, 
Intel,  SA,  ...) 

•  Guides  the  Analysis  Team  effort  during  DTE5 


BC  DCMP  Example 


Objective  Description 

Issue 

Number 

Issue  Description 

EEA 

Number 

EEA  Description 

MoM_Num 

MoM_Desc 

MC2 

Data 

RPWS 

Data 

Threads 

Which  Battle  Command 
System  best  supports  the 
3CE  force  development 
environment? 

1.1 

Does  the  BCS  support  scaling  of 
the  COP? 

1.1.1 

How  well  does  the  BCS  support 
scaling  of  the  COP? 

1.1. 1.1 

Level  of  difficulty  in 
scaling  the  COP 

X 

X 

X 

1.1. 1.2 

Flexibility  (by  echelon,  by 
BFA,  etc) 

X 

X 

X 

1.1. 1.3 

Ability  to  store  multiple 
pre-set  scaled  views 

X 

X 

X 

1.2 

How  robust  is  the  Intelligence 
Functionality  within  the  BCS? 

1.2.1 

How  well  does  the  system's 

Intelligence  Functionality  perform? 

1.2.1. 1 

Friendly  Info  (locations, 
status,  etc.) 

X 

X 

X 

1.2.1.2 

Automated  requirements 
management  and  asset 
visibility 

X 

X 

X 

1.2.1. 4 

Current  enemy  situation 
(fusion  of  ISR  and 
Intel/FS/AD/  Space 
sensors  data) 

X 

X 

X 

1.2.1.3 

Track  and  determine  ISR 
requirement  satisfaction 

X 

X 

X 

1.2.2 

How  well  does  the  user-interaction 
Intelligence  Functionality  perform? 

1. 2.2.1 

Friendly  Info  (locations, 
status,  etc.) 

X 

X 

X 

1. 2.2.2 

Current  enemy  situation 
(ISR  and  Intel/FS/AD/ 
Space  sensors) 

X 

X 

X 

1. 2.2.3 

Dynamic  adjustment  of 
the  ISR  synchronization 
plan 

X 

X 

X 
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Illustrative  Example  of  Thread  to  Support  Analytic 

Process 


Information  Requirements 


Deploy  Sensors 


Exploit 
Intelligenee: 
Make  Decision 
Deploy  sensors 
Fire  Mission 


Interaction  of 
All  Federate  Members 
Includes  man  in  the  loop 
data  input,  Development  of 
ISR  Requirements, 
analysis  and  evaluation 
of  information, 
and  exploitation 


C  N 

Update 

COP 

V _ / 


Produce, 

Disseminate 

Information: 


Receive 


Information: 

Fuse 

Analyze 

Evaluate 


Using  operational  threads,  such  as  ISR  (shown  above),  fires  and  SA 
dissemination,  in  varying  levels  of  load,  collect  on  applicable  MOMs. 
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Execution  CONOPS 


WHO,  WHAT,  and  WHERE 


UAMBL 


•  MC2 

•  RPWS 


ALCES  & 
NPST 


Task:  Collect  data  on  the  Battle  Command 
systems  through  observations  and 
demonstrations. 

Purpose:  To  identify  functional  capabilities 
and  limitations  of  the  M&S  system. 


Task:  Collect  data  on  the  communications 
effects  server  through  observations  and 
demonstrations. 

Purpose:  To  identify  functional  capabilities 
and  limitations  of  the  M&S  system. 


HUACHUCA 


ORION  & 
EMEW 


Task:  Collect  data  on  the  communications 
effects  server  through  observations  and 
demonstrations. 
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3CE  Comparative  Analysis  Lessons  Learned  (1  of  3) 


•  Planning:  the  systems  under  comparison,  as  well  as  the  required  capabilities, 
must  be  identified  up  front  and  have  the  buy-in  of  all  participating  Commands 

-  Must  be  an  integral  part  of  the  overall  planning  (i.e.,  technical  architecture,  federates, 
scenario,  data  collection  requirements,  and  support). 

-  Roles  and  responsibilities  associated  with  the  comparison  must  be  clearly  defined. 

-  Care  must  be  exercised  in  the  selection  of  models  and  systems  for  comparison  (i.e., 
avoid  the  comparison  of  dissimilar  systems). 

-  Must  identify  all  command-specific  requirements  for  each  of  the  functional  areas 
under  comparison. 

-  Systems  under  comparison  must  be  in  the  proper  places  and  thorough  integration 
testing  must  be  completed  prior  to  the  comparison. 

•  Data  Collection:  Must  support  technical,  functional  and  operational  requirements 

-  Must  have  a  central  data  repository  where  users  have  remote  access  and  community 
products  are  shared  (including  post-event  access). 

-  All  users  must  have  input  a  priori  to  the  data  repository  structure  to  ensure  their  data 
collection  and  analysis  needs  are  met. 

-  Must  have  an  integrated  database  system.  The  current  data  repository  segregates 
DIS,  HLA,  HLAM,  TENA  (ILH)  and  tactical  messages  into  separate  databases  making 
it  extremely  difficult  and  time  consuming  to  conduct  end-to-end  mission  thread 
analysis. 
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3CE  Comparative  Analysis  Lessons  Learned  (2  of  3) 


•  Operational  Analysis:  Must  develop  process  to  be  able  to  track  end-to-end 
threads 

-  Significant  challenges  arise  with  HLA  and  DIS  federations,  and  with  instability  of 
federations  (with  systems  up  and  down  it  leaves  gaps  that  do  not  support  analytical 
requirements) 

-  The  3CE  TOEL  is  not  conducive  to  operational  analysis  (end-to-end  threads);  events 
are  disjoint  and  do  not  represent  a  complete  operational  flow  of  information 

•  Technical  Stability:  Architecture  and  components  must  be  fully  operational 
during  testing 

-  Must  improve  the  stability  of  the  3CE  Federation  for  future  events.  The  instability  of 
the  federation  and  its  infrastructure  makes  it  impossible  to  isolate  what  works  and 
what  does  not  work,  and  produces  unacceptable  gaps  in  the  data  for  analytical 
purposes. 

-  Must  address  the  significant  challenges  that  arise  with  linking  HLA  and  DIS 
federations,  and  with  instability  of  the  federations  or  their  federates  (with  systems 
periodically  up  and  down  it  leaves  gaps  that  do  not  support  analytical  requirements). 
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3CE  Comparative  Analysis  Lessons  Learned  (3  of  3) 


•  Consistent  time  stamping  and  entity  ID  mechanisms  across  the  federation 

-  Entity  mappings  between  federates  (HLA  and  DIS)  was  not  completed  prior  to 
STARTEX;  resulted  in  MC2  icons  all  reflecting  Unknown 

-  Different  federations  within  the  3CE  environment  handle  entity  identification  differently 
(bumper  #’s,  URNs)  and  this  must  be  resolved 

•  Scenario 

-  Scenario/  TOEL  did  not  require  C2  systems  to  be  utilized  in  an  operational  mode 
(players  were  merely  following  TOEL) 

-  The  size  of  the  scenario  was  such  that  no  performance  benchmarks  could  be 
assessed  (570  entities  is  classified  as  a  light  load) 

-  The  scenario/  TOEL  must  support  comparison  requirements  (mission  threads,  load 
testing,  and  data  collection)  (i.e.,  events  were  disjointed  and  did  not  represent  a 
complete  operational  flow  of  information). 

•  Exercise  Support 

-  Must  have  more  robust  support  systems  (need  separate  technical/  testing  and 
Exercise  Control  (EXCON)  communications  assets,  and  federation  management 
tools) 

-  Need  more  green  suiters  on  the  systems  being  compared  and  for  use  as  SMEs 
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3CE  Comparative  Analysis  Process 
_ Recommendations  (1  of  3) _ 

•  Analytic  Methodology:  3-Phased  approach  should  be  maintained 

-  Must  do  better  job  of  obtaining  buy-in  on  connparison(s)  from  all  Commands 

-  Must  be  proactive  in  identifying  common  3CE  requirements,  sharing  exit  criteria  and 
products  (through  membership  on  appropriate  WG/CGs) 

-  Systems  being  compared  should  be  co-located;  static  and  dynamic  testing  preferred 

-  If  exit  criteria  are  not  met,  3CE  Management  must  be  briefed  and  corrective  actions 
must  be  agreed  upon  and  implemented  prior  to  proceeding 

•  Phase  I:  Determine  basis  of  comparison 

-  Must  do  better  job  of  identifying  valid  3CE  requirements  (need  consensus  from  all 
Commands):  should  use  PCS  ORD,  UFD,  ATEC  PCS  SEP  and  JC2  ORD/CDD 

-  Requirements  identification  must  be  worked  through  applicable  WG/CGs 

-  Operational  requirements  (mission  threads)  must  be  identified  early  on  and  data/ 
scenario  requirements  coordinated  with  applicable  WG/CGs  (Scenario  and  Data  at  a 
minimum);  mission  threads  must  be  conducted  with  the  environment  under  load 

-  Products  for  Phase  I  (issues,  EEA,  MoMs,  Data  Elements,  Draft  Analysis  Plan,  and  a 
Draft  Data  Collection  Management  Plan  (DCMP))  must  be  developed  ICW 
representatives  from  each  Command 

-  Exit  Criteria  for  Phase  I  must  be  fully  vetted  with  all  Commands  and  include: 

•  All  stakeholders  concur  that  their  users’  requirements  are  contained  in  the 
DCMP 

•  The  issues,  EEAs  and  MoMs  for  the  3CE  comparison(s)  are  adequate  to 
proceed  to  Phase  II 

•  All  stakeholders  concur  that  the  Analysis  Plan  and  the  DCMP  capture  their 
requirements  and,  when  successfully  executed,  will  provide  the  information 
needed  to  make  an  informed  decision  with  respect  to  the  comparison(s) 
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3CE  Comparative  Analysis  Process 
_ Recommendations  (2  of  3) _ 

•  Phase  II:  Verify  data  collection  capability  and  validate  analytical  approach 

-  Analysis  Team  ICW  DAT  WG/CG  must  verify  that  the  data  collection  mechanisms 
(and  data  repository  structures)  are  sufficient  to  meet  DCMP  requirements 
(especially  end-to-end  mission  thread  analysis);  if  they  are  not,  then  resolution  must 
be  reached  prior  to  commencing  Phase  III 

-  All  stakeholders  must  agree  that  the  event  scenario/  vignette  (and  TOEL)  supports 
the  analytic  approach  and  data  collection  requirements  (If  it  does  not,  then  changes 
must  be  made.) 

-  Products  for  Phase  II  (validated  analytic  approach  and  finalized  Analysis  Plan  and 
DCMP)  must  be  coordinated  with  representatives  from  each  Command 

-  Exit  Criteria  for  Phase  II  must  be  fully  vetted  with  all  Commands  and  include: 

•  All  stakeholders  concur  that  the  analytic  methodology  is  executable  and  that  the 
Analysis  Plan  and  the  DCMP  capture  all  of  their  requirements 

•  The  decision  to  proceed  to  Phase  III  is  contingent  on  achieving  the  Phase  II  Exit 
Criteria  and  successfully  competing  all  Integration  Tests 

•  Phase  III:  Conduct  data  collection  and  comparative  analysis 

-  Adequate  time  and  resources  must  be  dedicated  to  the  Technical,  Functional  and 
Operational  testing  outlined  in  the  DCMP 

-  Data  collected  and  archived  in  3CE  databases  must  be  readily  available  to  the 
Analysis  Team  (implies  distributed  access  for  all  stakeholders  until  the  analysis  is 
completed) 

-  Comparative  analysis  results  must  be  briefed  to  3CE  management  as  soon  as 
practical  (NLT  60  days  following  ENDEX) 
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3CE  Comparative  Analysis  Process 
_ Recommendations  (3  of  3) _ 

•  Post-Event  Actions/  Activities: 

-  Responses  to  the  questionnaire  developed  to  identify  business  processes,  policies, 
and  procedures  for  using  M&S  in  each  command  should  be  completed  and  returned 
in  a  timely  manner. 

-  Changes  must  be  made  that  will  allow  the  use  of  federation  output  data  to  support 
future  events: 

•  The  scenario/  TOEL  must  support  a  dynamic  operational  environment  so  that 
federation  output  data  can  be  used  to  support  the  analysis  of  mission  threads 
(Intelligence,  Situational  Awareness,  Fires,  and  Sustainment). 

•  The  manner  in  which  future  DTE  5  and  3CE  databases  are  built  must  support 
cross-walking  the  events  and  entities  associated  with  the  mission  threads. 

•  Access  to  simulation  output  data  must  be  made  available  for  post-event 
analysis  efforts. 

-  The  stability  of  the  federation  and  its  infrastructure  (DTE-MATREX  Bridge)  must  be 
improved  for  analysts  to  isolate  what  works  and  what  does  not  work,  and  to 
eliminate  gaps  in  the  data  for  analytical  purposes. 
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